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IMAGE PROCESSING METHOD AND APPARATUS 

BACKGROUND 
The invention relates to an image processing method and apparatus. 
Referring to Fig. 1, a typical computer system 5 may include a digital camera 12 
that electrically captures an optical image 13 and a computer 10 that may store, print or 
5 display the captured image. To capture the image, the camera 12 typically includes an 
image sensor 1 1 that captures digitized portions of the image (called pixels) and 
communicates indications (digital bits, for example) of these pixels to the computer 10 
(via a serial bus 14, for example). 



10 manner as the human eye. As a result, raw data that is provided by the image sensor 1 1 
may need to be transformed so that the transformed data indicates the optical image 13 as 
perceived by the human eye. However, before this transformation occurs, the raw data 
may need to be manipulated to compensate for other effects introduced by the camera 12, 
such as stray lighting effects, lens flare effects and the nonlinearity of the image sensor 

15 11. Also, the raw data may be manipulated to adjust a white color balance in the image 
that is indicated by the data. 

The term "white color balance" refers to a measure of the balance of colors in the 
captured image. For example, when an image is captured under a florescent light, the raw 
data from the image sensor 1 1 may indicate a generally green image. To correct an 

20 incorrect white color balance, the computer 10 may scale the pixel intensities (that are 

indicated by the raw data from the image sensor 1 1). For example, the computer 10 may 
scale the pixel intensities that indicate red, green and blue color components of the optical 
image 13 by different factors (called ocr, ccg> and ocb 5 respectively) to compensate for an 
white color imbalance. 

25 Unfortunately, an image processing circuit, such as the above-described camera 

12, may be specifically designed for a given image sensor and other components of the 



The image sensor 1 1 typically does not respond to light frequencies in the same 



1 



# • 



camera 12. Thus, there is a continuing need for an imaging processing circuit that more 
readily accommodates different camera components, such as image sensors that have 
different sizes and types, for example. 

5 SUMMARY 

In one embodiment, a method includes capturing an optical image to form raw 
data indicative of the optical image and using values in a look-up table to transform the 
raw data into transformed data that indicates a second image. A white color balance of 
the second image is computed, and the values in the look-up table are modified based on 
1 0 the computed white color balance. 

BRIEF DESCRIPTION OF THE DRAWING 
Fig. 1 is a schematic diagram of a computer system of the prior art. 
Fig. 2 is a schematic diagram of a camera according to an embodiment of the 
1 5 invention. 

Fig. 3 is an illustration of a look-up table of the camera of Fig. 2 according to an 
embodiment of the invention. 

Fig. 4 is a more detailed schematic diagram of the camera of Fig. 2 according to 
an embodiment of the invention. 
20 Fig. 5 is a flow diagram illustrating the processing that is performed by the 

camera according to an embodiment of the invention. 

DETAILED DESCRIPTION 
Referring to Fig. 2, an embodiment 20 of a digital camera in accordance with the 
25 invention includes a look-up table 26 that is used by the camera 20 to modify raw pixel 
data (that is provided by an image sensor 22) to compensate for such factors as the white 
balance of a captured image, nonlinearities introduced by the image sensor 22, and flare 
effects introduced by a lens 23. In this manner, the look-up table 26 stores values to 
which the different pixel intensities and colors that are indicated by the raw pixel data are 



2 



linearly mapped. This mapping, in turn, may compensate for one or more of the factors 
described above. More particularly, the table 26 may store intensity values in an array of 
storage locations, and each storage location may be uniquely addressed by a color 
component (a red, green or blue color component, for example) and the intensity of that 
5 color component. For example, for a red-green-blue (RGB) color space, the captured red, 
green and blue color pixels may be associated with three different groups 40, 42 and 44 
respectively, (see Fig. 3), of pixel intensity values in the look-up table 26. As an 
example, a red pixel (depicted as R(50) in Fig. 3) that has an intensity of fifty may map 
into a location of the table 26 that stores an intensity of 42 to produce a transformed red 

10 pixel intensity value (called R'(42) in Fig. 3) of 42. 

Instead of establishing the values of the table 26 in view of specific image sensor, 
lens and/or lighting conditions, the camera 20 may update the values of the table 26 in an 
iterative calibration process to optimize the values for the particular conditions and 
components being used in the camera 20. Thus, in a sense, the values of the table are 

15 self-adjusting to accommodate the non-ideal effects that are introduced by the camera 20 
and to accommodate the use of different sizes and types of components in the camera 20. 
In this manner, the camera 20 may initialize the table 26 with a set of values in an attempt 
to sufficiently compensate the raw pixel data that originates with the image sensor 22. 
The camera 20 may then analyze the image that is indicated by the transformed pixel 

20 data, determine if the white color balance of the indicated image is acceptable, and if not, 
change the values in the table 26 to improve the quality of the next image that is produced 
by the camera 20. 

The camera 20 captures and processes a particular optical image 18 in the 
following manner. The lens 23 and possibly other optics focus the optical image 18 onto 
25 the image sensor 22, and in response, the image sensor 22 furnishes signals that indicate 
the intensities of pixels of the captured image, i.e., indicates the raw data. If the image 
sensor 22 does not provide the red, green and blue (RGB) colors for each pixel location, 
then a color synthesis circuit 24 of the camera 20 may be used to interpolate the missing 
colors for each pixel location. For example, the image sensor 22 may provide pixel data 
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from which a Bayer pattern (for example) color synthesis may be used to interpolate the 
missing RGB colors for each pixel location to form three intensity values (for the three 
colors) for each pixel location. 

The resultant raw pixel data that is provided by the color synthesis circuit 24 (or 
5 image sensor 22, if the image sensor 22 provides true color pixel data) serves as indexes 
to point to the appropriate data in the table 26. The transformed data that is provided by 
the table 26 may then be processed by a color correction circuit 30 that transforms the 
pixel data so that the red, green and blue spectral responses of the indicated image match 
the corresponding spectral responses of the human eye. After this transformation, a color 

10 space conversion circuit 32 may convert the pixel intensities into a standard color space, 
such as a YCbCr color space, for example. From the data that is provided by the color 
space conversion circuit 32, a white color balance circuit 28 computes the white color 
balance of the image. 

In some embodiments, if the computed white color balance is outside of a 

1 5 predetermined range, then the white color balance circuit 28 changes the values in the 
table 26, and the camera 20 passes the captured frame through the above-described 
transformations again. In this manner, after determining that a particular image has an 
unacceptable white color balance, the white color balance circuit 28 may multiply the 
values of each group 40, 42 and 44 (see Fig. 3) of the table 26 by associated scalars a R > 

20 etc* and ocb, and subsequently, the new values in the table 26 may be used in the next set 
of transformations. 

Thus, in some embodiments, the circuitry of the camera 20 forms a feedback loop 
that may be used in an iterative process to compensate for the white color balance, as the 
camera 20 permits processing of the image to account for other camera-introduced non- 
25 ideal effects before attempting to modify the values of the table 26 to readjust the white 
color balance. In some embodiments, when the camera 20 is in a still capture mode, the 
camera 20 may process the still image in the above-described feedback loop to adjust the 
values in the table 26 until the white color balance is acceptable. In other embodiments, 
the camera 20 may use a predetermined number (two, for example) of passes through the 
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feedback loop. Thereafter, as long as the camera 20 is turned on, the camera 20 may 
periodically check (every ten frames, for example) the white color balance to determine if 
the white color balance is in a predetermined range, as the lighting conditions (one of the 
main variable) may remain substantially the same over a small number of frames. 
5 For video, in some embodiments, the camera 20 may permit each frame to pass 

through even if the white balance is unacceptable, as the camera 20 may make corrections 
to each successive future frame until the white color balance is properly adjusted. Once 
adjusted, the camera 20 may periodically check the white color balance (via the white 
color balance circuit 20) during selected frames of the video. 

10 Among the other features of the camera 20, in a bypass mode, the camera 20 may 

include a bypass path 39 for communicating the raw pixel data directly from the image 
sensor 22 to a computer 290. An edge enhancement circuit 34 may receive pixel data for 
an outgoing frame from the color space conversion circuit 32 and modify the data to 
further emphasize edges of the image to improve the image's contrast. A compression 

15 circuit 36 may compress the pixel data that is provided by the edge enhancement circuit 
34 to reduce the bandwidth used to communicate a particular frame to the computer 290. 
Instead of communicating the frame to the computer 290, the frame may be stored (at 
least temporarily) in a memory 37. 

Referring to Fig. 4, as a more specific example, the camera 210 may include a 

20 capture and signal processing unit 248 that may interact with the image sensor 22 to 

capture the optical image and transfer a frame of data that indicates the resultant raw pixel 
data to a random access memory (RAM) 263. To accomplish this, the capture and signal 
processing unit 248 may be coupled to a bus 220, along with a memory controller 261 
that receives the frame from the bus 220 and generates signals to store the data in the 

25 RAM 263. Indications of the look-up table 26 may also reside in the RAM 263. A 

processor 262 may access the data in the RAM 263 to perform, for example, the color 
synthesis, look-up table transformation, color correction, color space conversion, white 
color balance computation, and edge enhancement functions. The processor 262 may be 
coupled to the bus 220 via a bus interface 270. In this context, the term "processor" may 
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generally refer to one or more microprocessors, such as a microcontroller, an X86 
microprocessor, an Advanced RISC Machine (ARM) microprocessor or a Pentium® 
microprocessor, as just a few examples. 

Among its other features, the camera 20, the camera 20 may include a 
5 compression unit 268 that may interact with the RAM 263 to compress the size of the 
processed frame before storing the compressed frame in the memory 37, such as a flash 
memory 278. To accomplish this, the compression unit 268 may be coupled to the bus 
220, along with a flash memory controller 274 that receives the compressed frame from 
the bus 220 and generates signals to store the data in the flash memory 278. To transfer 

10 the compressed frame to the computer 290, the camera 20 may include a serial bus 

interface 266 that is coupled to the bus 220 to retrieve the compressed frame from either 
the RAM 263 or the flash memory 278. The serial bus interface 266 may generate 
signals on a serial bus 280 (a Universal Serial Bus (USB), for example) to transfer an 
indication of the compressed frame to the computer 290. The USB is described in detail 

15 in the Universal Serial Bus Specification, Revision 1.0, published on January 15, 1996, 
and is available on the Internet at www.intel.com. The camera 20 may also include a 
read-only memory (ROM) 269 that may be coupled to the bus 220. The ROM 269 may 
store program 170 that causes the processor 262 to perform the above-described functions 
when the processor 262 executes the program 170. 

20 To summarize, the camera 20 may use the following technique 300 (depicted in 

Fig. 5) to process an optical image captured by the camera 20. In particular, the camera 
20 may capture (block 302) an optical image, interpolate (block 304) any missing pixel 
colors from the captured image, select (block 306) values for the look-up table 26 and 
then perform an interactive process to adjust the values in the look-up table 26. In this 

25 manner, the camera 20 may perform (block 308) a transformation on the raw pixel data 
via the look-up table 26 and then perform (block 310) color correction. Next, the camera 
20 performs (block 312) color space conversion before determining (diamond 314) 
whether the white color balance is acceptable. If not, the camera 20 then readjusts (block 
316) the values in the look-up table 26 and then returns either to block 308 or 
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alternatively to block 302, depending on the particular embodiment. If the camera 20 
determines that the white color balance is acceptable, then the camera 20 returns to the 
block 302 to capture another optical image. 

Other embodiments are within the scope of the following claims. For example, 
5 although a camera is described as an image processing circuit in accordance with the 
invention, other image processing circuits (a scanner, for example) may embody the 
invention. 

While the invention has been disclosed with respect to a limited number of 
embodiments, those skilled in the art, having the benefit of this disclosure, will appreciate 
10 numerous modifications and variations therefrom. It is intended that the appended claims 
cover all such modifications and variations as fall within the true spirit and scope of the 
invention. 
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